Automated detection of bird roosts using NEXRAD radar data and Convolutional Neural Networks

NEXRAD radar

The NEXRAD radar network comprises 151 Doppler weather surveillance radars. These radars complete a series of rotational scans every 5–10 min. They scan the atmosphere at different tilts or elevations, however for this paper we only focus on the lowest elevation scan (i.e. 0.5°). Radar data used for this research came from the level 2 NEXRAD radar archive, which is publicly accessible via Amazon Web Services.1 This dataset extends back to the mid‐1990s and contains data from single‐polarization Doppler radars (which we refer to as legacy radar in this paper) and dual‐polarization Doppler radar. Data from legacy radars include three products: (1) Reflectivity, which is a measure of the reflected energy from objects within a given air volume; (2) Doppler Radial Velocity, which indicates the relative movement of objects toward or away from the radar; and (3) Spectrum Width, which is variability of the mean radial velocity.

During 2012 and 2013, all NEXRAD radars were upgraded to dual‐polarimetry, which means that they now transmit and receive radio pulses that are polarized in the vertical and horizontal orientations. This capability allows for better assessment of the shapes of objects detected by radar, and dual‐pol radars offer three new data products in addition to the legacy products: (1) Differential Reflectivity (ZDR), which indicates the difference in reflectivity between the horizontal and vertical pulses; (2) Differential Phase (φ_DP), which is a measure of the difference between horizontal and vertical pulse phase shifts; and (3) Correlation Coefficient (ρ_HV), which assesses the similarity between the behaviors of the horizontally and vertically polarized pulses within a pulse volume.

Spectrum Width and Differential Phase (φ_DP) are generally not considered to be useful for detecting bird roosts, so we did not use them as model inputs. Hence used reflectivity, radial velocity, differential reflectivity and correlation coefficient as model inputs. Reflectivity, or echo intensity, provides an overall view of airborne objects and has been shown to be useful for detecting bird roosts as well as calculating the density of birds (Diehl and Larkin 2005). Radial velocity is often useful for identifying birds as it can reveal when airborne objects move in opposition to air currents (Gauthreaux and Belser 1998). Differential Reflectivity (ZDR) often reveals distinctive asymmetries in the reflectivity of biological targets (Stepanian and Horton 2015). Finally, correlation coefficient (ρ_HV) is typically lower for biological echoes as opposed meteorological echoes (Van Den Broeke 2013), and it has also been used to determine the orientation of flying birds (Stepanian and Horton 2015).

Roost data

Swallow roosting occurs in the late summer months, and roosts are most apparent in radar data early in the morning, from 20 min before to 40 min after sunrise, when the birds typically leave the roosting site. Kelly and Pletschet have manually identified and mapped hundreds of bird roosts through exhaustive searches of years of radar data from 64 different NEXRAD Radars (Kelly and Petschet 2018). Their search protocol involves searching examining radar imagery from one hour before local sunrise until 30 min after local sunrise from June 1 to September 30 (Kelly and Pletschet 2018), an effort that requires examination of 70,000–140,000 radar images per year.

The information used to train our machine‐learning system came in part from a subset of the roost data derived by Kelly and Pletschet (2018), which underwent a post processing step to manually identify which particular radar scans had roosts visible. We also used data generated by an interactive web‐page2 that was used to collect labels for previous research projects (Laughlin et al. 2013, 2016). This data is part on an ongoing database that contains labeled radar data with information about the presence and locations of bird roosts. (Laughlin et al. 2014). The final set of manually labeled roost data came from 10 different radars: KAMX, KBRO, KDOX, KGRK, KJAX, KHGX, KLCH, KLIX, KMLB and KMOB and was a mix of legacy and dual‐polarimetry data. A distribution of labels by dataset as well as legacy and dual‐pol data can be seen in Figure 1. Both of the datasets contained primarily positive labels, with few instances where it was clear that a roost was not found. To increase the number of negative labels, we selected radar scans from 2 to 1 h before sunrise and from 1 to 2 h after sunrise, leaving a 2 h window in between. The noise in our radar images (dust, weather, sun‐streaks, etc.) appeared to be similar directly before, during and after the roost is visible, which should ensure that our machine learning algorithms are detecting roosts as opposed to other patterns in the data to make classifications.

Once the desired radar scans were identified, each was acquired from the AWS database and converted from radial coordinates to two‐dimensional raster images. We use the Py‐Art library to create the images of the radar products (Helmus and Collis 2016). We used only the lowest radar tilt from each scan (0.5° of elevation), which is where most bird activity is evident. The reflectivity, velocity, ρ_HV and ZDR radar products were all saved as individual images. These images serve as the input to our machine learning models.

Table 1 shows how many training labels we have as inputs to our model.

Machine learning methods

We employed three general machine learning approaches to work toward optimizing roost detection in terms of accuracy and computation time. The first approach used a relatively simple traditional, feed‐forward artificial neural network (ANN) as depicted in Figure 2. The second approach used a sophisticated convolutional neural network–specifically the ‘Inception’ network created by Szegedy et al. (2016)–as a starting point and modified the last two layers to tailor it for roost identification. The third approach developed a shallow convolutional network from scratch, with only two convolutional layers as depicted in Figures 3 and 4.

We processed each of the four types of radar imagery (reflectivity, radial velocity, differential reflectivity and correlation coefficient) separately using each machine learning approach as shown in Figure 5. When only legacy data were available, only two types of imagery (reflectivity and radial velocity) were used. In the final step, the results from each of the four networks was combined within a dense neural network to generate an aggregate classification for each image (either containing a roost or not). One advantage of training the networks on the radar fields separately instead of together is that is reduces the number of input variables a single network is required to train on. Our design was inspired by an approach that was used successfully to train separate convolution layers in parallel, each on a different image rotation (Dieleman et al. 2015). The results were then fed into dense layers of the network (Dieleman et al. 2015).

To improve the speed and accuracy of our ANN and Shallow CNN we employed batch normalization (Ioffe and Szegedy 2015). For the transfer learning comparison we chose a network that also used batch normalization, the Inception‐v3 network (Szegedy et al. 2016). Batch normalization employs an additional step within each node to normalize their outputs over a batch of images, which stabilizes the node outputs and hastens convergence toward useful sets of weights. Batch normalization was first introduced in 2015 and it improved the accuracy of ImageNet classification while simultaneously speeding up learning 14 times (Ioffe and Szegedy 2015). All the networks were trained using binary cross‐entropy as the loss function, and we used average accuracy as the metric to see how well training progressed.

Our traditional ANN employed multiple layers of artificial neurons or nodes. In a machine‐learning context, a node combines a vector of inputs (which might be pixel values) with a corresponding vector of weights to generate a summation, or output, that gets transformed by an activation function and passed to the next layer in the network. ANNs consist of highly connected layers, and the weighted connections among the nodes change as the network is trained to recognize the input data (Mitchell 1997). For image classification problems, ANNs have an input layer that takes in pixel values for an image, a number of hidden layers that change the weights, and an output layer with a number of nodes equal to the number of classes (Driss et al. 2017). The ANN we used had an input layer, three hidden layers, and an output layer, and it worked on gray‐scale images. The input layer accepted the 240 × 240 images and connected them to subsequent layers. The three hidden layers consisted of 128, 64 and 8 nodes. Each of these layers used batch normalization and a rectified‐linear‐unit activation function. The output layer, which had only two nodes, and used the softmax activation function to generate probabilities for the two possible classifications.

As a second approach, we used Convolutional Neural Networks (CNNs). Convolutional networks use kernels (i.e. arrays or matrices of weighting values) that are applied across an image to identify features that contribute to image classification. In 2012, Deep CNNs achieved record breaking results for classifying the thousands of annotated images that comprise the ImageNet dataset (Krizhevsky et al. 2012). Subsequent work has expanded on CNNs using varying architectures such as VGG16, GoogLeNet, Inception‐v3 and ResNet to improve CNN results for classifying images (Simonyan and Zisserman 2014; Ioffe and Szegedy 2015; Szegedy et al. 2015, 2016; He et al. 2016). For this paper we chose to use the Inception‐v3 network. Although CNNs produce robust results, they require large amounts of training data to be effective since the network has to modulate millions of parameters (Oquab et al. 2014).

Because we lacked sufficient data to fully train a deep CNN from scratch, we employed a transfer‐learning approach, wherein foundational knowledge learned on one dataset is applied to a new dataset (Oquab et al. 2014). As an initial foundation for our CNN we used Inception‐v3 (Szegedy et al. 2016), a network trained on ImageNet (Deng et al. 2009), to make 1000 image classifications. The initial nodes in the Inception‐v3 network have learned to identify general features (e.g. edges, shadows and curves) that can be applied generically to other image data (Shin et al. 2016). We initialize our model with the weights learned during the feature extraction part of training the Inception‐v3 network in ImageNet. We then replace the classification part of the model (the last two layers) in order to perform 2‐way classification instead of 1000‐way classification. The last two layers consist of a fully connected layer and a softmax output layer. The weights of all except the very last two layers of the Inception‐v3 network are frozen during training, and only the last two layers were fine tuned using the novel radar dataset (Shin et al. 2016). This transfer‐learning approach allows us to train deeper CNNs with smaller training datasets as there are fewer weights and parameters that the network is required to optimize. A full description of the Inception‐v3 network can be found in Szegedy et al. (2016). The CNNs we employed used full color images for each radar product that were derived from a standard NEXRAD color scheme implemented in Py‐ART (Helmus and Collis 2016). Color images were used because the Inception‐v3 CNN was originally trained on color image inputs.

Our third approach trained a shallow CNN from scratch. The shallow CNN had only two convolution layers and one fully connected layer in the network. As with the Inception‐v3 CNN, the shallow CNN used RGB channel values from each radar product as input. The first convolution layer employed 32 filters, a kernel size of 5, batch normalization and rectified linear units (as an activation function). The second convolution layer has the same setup except with 64 filters. After each convolution layer, we down‐sampled the data using max pooling with a pool size of 2 and a stride size of 2.

Metrics

We used four different metrics for evaluating our machine learning results. We evaluated the total accuracy (ACC), the true positive rate (TPR), the true negative rate (TNR) and the area under the receiver operating characteristics (ROC) curve (AUC). ACC, TPR and TNR can all be calculated using the number of true positives (TP), true negatives (TN), false positive (FP) and false negatives (FN).

(1)

(2)

(3)

A ROC curve can be used for visualizing a classifier's performance and the AUC can be used to compare different ROC curves (Fawcett 2006). AUC values range from 0 to 1 where 1 shows a perfect classifier and a score of 0.5 represents random guessing (Fawcett 2006). An AUC value of .9 or above is considered to be a good result. For more details on how to calculate the AUC see Fawcett (2006).

Model training and validation

Establishing a machine‐learning classifier typically involves training, validation and testing sets to determine whether the trained models have arrived at a robust solution (Cohen 1995). Because our classification models combined inputs from multiple radar products, our training, validation, and testing was implemented as a two‐step process. First the models assigned a classification probability to each radar product separately. Then these classification probabilities served as the inputs to a second classification layer. Ideally these two machine‐learning steps would be trained with two different validation sets. However, we did not have a sufficient number of labels to take this approach. Therefore we split the data into three different groups that we will refer to as A, B and C. Group A contained 60% of the data and the remaining 40% were split equally between groups B and C. When training the models to detect bird roosts from image data for a single radar product we use A as our training set, B as our validation set, and C as our testing set. For the second stage of learning, we input the probabilities of the different radar products into a hidden layer for the aggregate classifier. At this stage of the problem we use B as our training set, A as the validation set, and C as the test set. We swap A and B in order to get a second validation set as well as give the network data to train on that will be as similar to the test set as possible. C is consistently used as the test set throughout. In each of these cases the entire dataset is used either in train, test, or validation.

To assess confidence in our final classifications we used a K‐fold cross‐validation, wherein the data were partitioned into k subsets (folds), with training performed on k−2 folds, validation performed on 1 fold and testing performed on the remaining fold (Kohavi 1995). In our case, we put 3 folds into set A, 1 fold into set B and 1 fold into set C. The training is repeated k times, where each fold is used as the testing and validation fold exactly once. K‐fold allows us to evaluate every labeled datum. We used 5‐fold cross‐validation to train and evaluate our models. We chose a small number for k because convolutional neural networks are computationally expensive to train.

For each of our metrics we calculated the confidence interval using the bootstrapping percentile method. The percentile method calculates the chosen metric (e.g. loss or accuracy) on randomly selected samples of the data iteratively (Efron and Tibshirani 1986). Then for a 95% confidence interval we take the upper and lower 2.5% points of distribution (Efron and Tibshirani 1986). This is a range that 95% of the bootstrapped samples fall within. The upper and lower bound of the distribution become the confidence interval for the performance metric (Efron and Tibshirani 1986).

Each testing fold was evaluated using its corresponding network. The results from each of the testing folds were then combined. To compute the confidence intervals for ACC and AUC, we randomly selected one thousand samples with re‐sampling from the combined testing results. For TPR we select one thousand samples from the roost data and for TNR we select one thousand samples from the no roost data. We repeat this process for one thousand iterations on each of our metrics in order to compute the confidence intervals.

Future work

In the future, we hope to fully automate the bird roost detection process by preprocessing the radar images. Just as radar data is quality‐controlled for weather (Lakshmanan et al. 2014), we could filter out weather from our radar data to eliminate some of the noise from our radar data. In addition, biological reflectivity generally falls within a range of −10 dBZs to 10 dBZs (Koistinen 2000), and by filtering out values outside this range we can eliminate some of the noise. We cannot use a reflectivity filter to fully determine where birds are since light drizzle and insects are often detected in this range as well (Koistinen 2000). Biological scatter will likely have a high differential reflectivity and a low correlation coefficient (Van Den Broeke 2013), and we could use these properties to further filter and clean the data.

We are also not currently taking advantage of the temporal component of the data during learning. The expanding roost rings over sequential radar snapshots are an important roost characteristic used in manual detection of bird roosts. There are several machine learning methods such as Recurrent Neural Networks (RNN) or Long Short Term Memory networks (LSTM) that use temporal data. LSTM networks (Donahue et al. 2015) have been used on sequences of images, for example to re‐identifying a person over disjoint cameras (Wu et al. 2016) or to detect the type of activity (run, jump, etc.) a person is performing in a video (Yeung et al. 2018). An LSTM network is one way to potentially increase accuracy using temporal data, although it's worth mentioning that they can require more labeled data since they need to learn a larger amount of parameters. They can also take longer to train, although this should not be an issue for a small network like the shallow CNN.

Our results could be improved with additional Dual Polarization Labels. As stated above, CNNs require lots of data to train properly (Oquab et al. 2014). Our dual polarization radar results were better than our legacy radar results even though we had fewer dual polarization machine learning inputs. Hand classifying roost data is a time consuming process, however, it would be useful for better automated roost detection. One of the advantages of polarimeter radar for weather is that it helps quality control the biology more accurately from the weather data (Zrnic and Ryzhkov 1998). It stands to reason that the same method that is used to remove biology from the radar data can be used to find it as well.

The biggest next step for this project is locating the bird roosts within the radar images instead of only detecting them, which is a very challenging problem. Image segmentation or Regional‐CNNs are both potential approaches to this research. There is a trade‐off between recognizing and locating objects within an image (Maggiori et al. 2017), so having a network that can detect roosts first will prove useful for the next stage of this research.